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DETAILED ACTION 

1. A request for continued examination under 37 CFR 1.114 was filed in this application after 
appeal to the Board of Patent Appeals and Interferences, but prior to a decision on the appeal. Since this 
application is eHgible for continued examination under 37 CFR 1.114 and the fee set forth in 37 CFR 
1.17(e) has been timely paid, the appeal has been withdrawn pursuant to 37 CFR 1.1 14 and prosecution in 
this application has been reopened pursuant to 37 CFR 1.1 14. Applicants submission filed on 02/22/05 
has been entered. Claims 25-53 remain pending. 

2. The finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. 
Applicant's submission filed on 02/22/05 has been entered. 

3. Applicant's arguments filed 02/22/05 have been fully considered but they are not persuasive. 
Arguments have been fully addressed on the response to arguments section below. 

Claim Rejection under USC 101 

4. Claims 25-53 are rejected under 35 U.S.C. § 101 which reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, 
or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of 
this title. 

5. Method claim 25-53 are rejected under 35 U.S.C. §101 because the claimed invention is directed 
to non-statxitory subject matter. 

In this case, computer-related inventions whether descriptive or functionally descriptive material 
are non-statutory categories when claimed as descriptive material per se (see Warmerdam, 33 FJd at 
1360 USPQ2d at 1759), falling under the "process" category (i.e. inventions at that consist of a series of 
steps or acts to be performed). See 35 U.S.C. 100(b) ("The term process means, art, or method, and 
includes a new of a known process, machine, manufacture, composition of matter or material"). 
Functional descriptive material: "data structures" representing descriptive material per se or computer 
program representing computer listing per se when embodied in a computer-readable media are still not 
statutory because they are not capable of causing functional change in the computer. However, claimed 
computer-readable medium encoded with a data structure defined structural and functional 



Application/Control Number: 09/495,175 (BENITEZ, et. al.) 
Art Unit: 2142 



Page 3 



interrelationships between the data structure and the computer software and hardware component, which 
permit the data structure's fijnctionality to be realized, and is thus statutory (see MPEP 2106). 

The method claims 35-53 do not seem to be described as being implemented in any tangible 
and/or limited to any tangible embodiment(s) (e.g. hardware components) in view of Applicant's 
disclosure. As such, the claim is not limited to statutory subject matter and is therefore non-statutory. To 
overcome this type of rejection the claims need to be amended to include only the physical computer 
media or embodied on computer readable media medium, e.g. the computer-readable recording medium 
storing the program for performing the method, etc. 

Claim Rejection under 35 USC §103 

6. Quotation of 35 U.S.C. § 103(a) which forms the basis for all obviousness rejections set forth in 
this Office action may be found in previous office action. 

7. Claims 25-53 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sezan et. al. (Sezan) 
MPEG-7 Standardization Activities in view of Bergman et. al. (Bergman) U.S. Patent No. 6,564,263 Bl. 

Regarding claim 25, Sezan teaches features of the invention substantially as claimed, teaching a 
system/method comprising: 

identifying multimedia categories (types) from received multimedia contents (introduction 
section, page 517); 

extracting multimedia objects to generate multimedia object descriptions from the multimedia 
content for a multimedia (type) (feature extraction, page 518, descriptors section page 518, Fig. 1, 
description generation); 

generating, from the multimedia object descriptions (description, page 518), an entity relationship 
representation (graph descriptions) for a multimedia type (features hierarchy section, page 519); however 
prior art does not explicitly teach integrating the multimedia object descriptions and entity relation graph 
descriptions to generate a description record to represent content embedded within the multimedia 
content; 

Bergman teaches aggregating, incorporating or combining, i.e. integrating from the multimedia 
object descriptions (col 12/lines 17-26), entity relation graph descriptions (col 3/lines 27-36, 46-51, 59- 
62, Figs. 3-7, col 4/lines 20-30, integrated multimedia descriptions and entity relationships descriptions: 
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col 6/lines 57-64, col 8/lines 42-67, entity-relationships, col 9/lines 55-col 10/line 10) to generate an 
object ("description record"); 

wherein a composite object supports embedding of multimedia contents (col 7/lines 2-6, 
embedded hyper-linking, col 15/lines 4-17, spatial relationship of object within/contained, col 17/lines 
10-19, object within another object, col 17/lines 49-52); additionally teaching 

identifying multimedia type or modalities in multimedia content (col 3/lines 21-36, description 
data type, col 12/lines 23-64, select identified type: col 19/lines 41-49); 

capture (extract) multimedia object features, attributes or modalities from multimedia objects to 
form multimedia object descriptions and entity relation descriptions from the multimedia object (coi 
3/lines 37-51, col 6/lines 39-43, extract: col 19/lines 54-58); 

generating entity relationships descriptions and multimedia object descriptions from a multimedia 
category, type or modality (develop: col 14/lines 44-48, create: col 12/lines 43-50, generate: col 19/lines 
50-53); 

generating "non-hierarchical entity relation graph descriptions", i.e. general relationships 
descriptors among multimedia objects (spatial relationship of object within/contained, col 17/lines 10-19, 
object within another object, col 17/lines 49-52, including objects with spatial, temporal relationships and 
thus allows the description of these relationships between presentation object col 3/lines 37-51, see Fig. 5- 
6 and using a graphic block-based representation to describe spatial and temporal relationships between 
the objects col 5/lines 64-col 6/line 14). 

It would have been obvious to one ordinary skilled in the art at the time the invention was made 
to include Bergman's teachings for aggregating, incorporating or combining, i.e. integrating the 
multimedia object descriptions and their respective entity relation graph descriptions to generate a 
description record to represent content embedded within the multimedia content, motivation would be to 
generate entity relation description based on the multimedia object descriptions for multiple multimedia 
content types including a composite multimedia object description that represents content embedded 
within the multimedia content, as taught by Bergman. 

Regarding claim 26, multimedia object pyramid (hierarchy) descriptions for one of the multimedia types 
(Figs. 3-4 & 8, hierarchical model of multimedia object descriptions based on content type; col 8/lines 12- 
16, 20-67, Sezan: feature hierarchy, page 519). 

Regarding claim 27, the multimedia types includes image (Sezan: media type, page 519). 
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Regarding claim 28, separating (segmenting) each multimedia content into descriptor defining portions 
(segments) including content from one of the multimedia type for the multimedia content (Sezan; page 
518); and 

generating one feature representing a feature for one of the portions by feature extraction and 
annotation (Bergman: feature description: col 8/lines 42-46, annotation: col 8/lines 55-58, extraction col 
3/lines 37-51); 

wherein the generated multimedia object descriptions comprises one feature description for one 
segment (Bergman; col 8/lines 20-41). 

Regarding claim 29, wherein the segments are selected from the group consisting of local segments and 
global segments (Bergman: col 8/lines 55-58). . 

Regarding claim 30, feature description from the group consisting of media, semantic and temporal 
features (Bergman: col 8/lines 42-67). 

Regarding claim 31, a feature description selected from the group consisting of data location, scalable 
representation and modality trans-coding (Bergman: col 9/lines 25-37). 

Regarding claim 32, wherein the semantic features are further defined by one feature description selected 
from the group consisting who (Bergman: col 8/lines 47-50). 

Regarding claim 33, temporal features are further defined by one feature description consisting of 
duration (Bergman: objects duration, col 16/lines 16-34). 

Regarding claim 34, the applied prior art further teaches generating media object descriptions from the 
multimedia segment for one of the multimedia types by media object extraction processing (Sezan: 
feature extraction, page 518, descriptors section page 518, Fig. 1, description generation, Bergman: 
capture (extract) multimedia object features, col 3Aines 37-51); 

generating media object hierarchy descriptions from the generated media object descriptions by 
object hierarchy construction and extraction processing (Bergman: col 14/lines 44-48; Sezan: description, 
page 518, features hierarchy section, page 519); and 
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generating media entity relation graph descriptions from the generated media object descriptions 
by entity relation graph generation processing (Bergman: col 3/lines 27-36, 46-51, 59-62, Figs. 3-7, col 
4/lines 20-30, col 6/lines 57-64, col 8/lines 42-67, entity-relationships, col 9/lines 55-col 10/line 10). 
Regarding claim 35, segmenting the content of each multimedia type in the multimedia object into 
segments within the multimedia object by media segmentation processing (Bergman: col 3/lines 21-36, 
col 12/lines 23-64; Sezan: categories (types) introduction section, page 517); 

generating one feature description for one of the segments by feature extraction and annotation 
(Bergman: feature description: col 8/lines 42-46, annotation: col 8/lines 55-58, extraction col 3Aines 37- 
51); 

wherein the generated media object descriptions comprise the feature description for one of the 
segments (Bergman; col 8/lines 20-41). 

Regarding claim 36, substantially the same as claim 30, same rationale of rejection is applicable. 

Regarding claims 37-40, wherein generating media object pyramid (hierarchy) descriptions generates 
terminal/composite objects define as multimedia content pyramid (hierarchy) descriptions of the media 
object descriptions (Bergman: col 3/lines 27-36) based on relationships of media objects represented by 
the media object descriptions (Bergman: col 3/lines 41-51), and wherein the relationships consisting of 
media feature relationships, semantic feature relationships, temporal feature relationships, and spatial 
feature relationships (Bergman: col 6Aine 1-14, 57-67, col 7/lines 10-14, 20-25). 

Regarding claims 41 and 46, wherein generating media entity relation graph descriptions (Bergman: col 
9/lines 55-col 10/line 10) generates entity relations graph descriptions of the media object descriptions 
based on entity-relationships and dependency-entity relationships of media objects represented by the 
media object descriptions (Bergman: col 14/lines 45-48, col 1 9/lines 50-53), 

wherein the relationships are selected from the group consisting of media feature relationships, 
semantic feature relationships, temporal feature relationships and spatial feature relationships (Bergman: 
col 6/lines 1-14, 57-67, col 7/lines 10-14, 20-25). 

Regarding claims 42-45, wherein generating multimedia object pyramid (hierarchy) descriptions 
generates multimedia object hierarchy descriptions of the multimedia object descriptions based on media 
content relationships of multimedia terminal/composite objects represented by the multimedia object 
descriptions (Bergman: col 11/line 27-33, inter-feature relationships: col 1 9/lines 50-53, inter-object 
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relationships, col 15/lines 10-15); based on temporal, spatial feature relationships of multimedia objects 
(Bergman: col 15/lines 10-15); and semantic feature relationships of multimedia object (Bergman: col 
6/lines 58-64). 

Regarding claim 47, receiving (Sezan: page 517) and transcoding (encoding) the multimedia object 
descriptions into encoded description information (Bergman: col 7/lines 20-24), and storing the encoded 
description information as one (description) record (Sezan: left column, page 518). 

Regarding claim 48, this claim combines limitation(s) substantially the same as claims 25, and 47, same 
rationale of rejection is applicable 

Regarding claims 49 8l 50, the encoding comprises binary encoding (Bergman: col 13/lines 31-33). 

Regarding claims 51 & 52, the encoding comprises the extensible Markup Language (XML) encoding 
(Bergman: col 14/lines 4-18). 

Regarding claim 53, includes limitations substantially the same as claim 25, same rationale of rejection is 
applicable, and further generating, from the multimedia object descriptions, multimedia object hierarchy 
descriptions (Bergman: generating entity relationships descriptions and multimedia object descriptions 
from a multimedia category, type or modality develop (col 14/lines 44-48, create: col 12/lines 43-50, 
generate: col 19/lines 50-53) by object hierarchy construction and extraction processing, for a multimedia 
content modality or feature (type) (extract: col 3/lines 41-51, col 6/lines 39-42, construct: col 12/lines 43- 
50); and 

integrating the multimedia object descriptions and the multimedia object pyramid (hierarchy) 
descriptions to generate a descriptor (description record) to represent content embedded within the 
multimedia content (Bergman: integration, col 12/lines 17-26, entity relation graph descriptions, col 
3/lines 27-36, 46-51, 59-62, Figs. 3-7, col 4/lines 20-30, integrated multimedia descriptions and entity 
relationships descriptions: col 6/lines 57-64, col 8/lines 42-67, entity-relationships, col 9/lines 55-col 
10/line 10) to generate a composite object); 

wherein a composite object supports embedding of multimedia contents (col 7/lines 2-6, 
embedded hyper-linking, col 15/lines 4-17, spatial relationship of object within/contained, col 1 7/lines 
10-19, object within another object, col 17/lines 49-52). 



Application/Control Number: 09/495,175 (BENITEZ, et. al.) 
Art Unit: 2142 



Page 8 



Claim Rejection under 35 USC §103 

8. Claim 25 as amended is rejected under 35 U.S.C. 103(a) as being unpatentable over International 
Organization of Standardization, Organization International Normalization, ISO/IEC JTC1/SC29AVG11, 
Coding of Moving Picture and Associated Audio, MPEG 98 (MPEG-7 Evaluation Process Document), 
Atlantic City, October 1998 p. 1-75 (ISO hereafter) in further view of The Characteristics of Digital 
Video and Considerations of Designing Video Databases, Chang, C, Lin, K. & Lee, S., ACM 0-89791- 
812-6/95, 1995, p. 370-377 (Chang hereafter) 

Regarding claim 25, ISO discloses: defining descriptors (Ds) in a chosen (identified) multimedia type (p. 
66); 

extracting multimedia object description from the multimedia content for one type (section 5.3, p. 
20, which type of media to use to extract your descriptor, see annex E, p. 34); 

defining (generating) from the multimedia object description features descriptor (Annex E, p. 34, 
defining features, e.g. syntax, semantics, etc., p. 31, p. 66); 

generating from multimedia objects descriptions entity relational graph descriptions for the 
multimedia type (relationship between description and data supplying a model... p. 66, relationships 
within a description scheme (DS) and between descriptors (Ds) to represent spatial, temporal, structural 
and conceptual relations p. 17, graphical representation for description schemes using a UML notation, p. 
31, for generating diagram, p. 72, diagrams are graph describing the content media, p. 71, Diagram (A1.2) 
shows an non-hierarchical entity relation graph, p. 75), although discloses generating object descriptors 
for describing content features within a content object, including the generation of multimedia object 
descriptor and multimedia object relationships descriptors including it entity-relationships graph, it does 
not disclose a storage structure (at least one record) combining these descriptors; 

Chang teaches object descriptors comprising attributes and attribute values for given descriptive 
representation of content objects for describe content within the content objects, attributes having 
different data types for the different content types, e.g. text, image, video, etc. (right column on p. 374); 

a storage structure for storing definition of schemas, the attributes of features and the instances of 
objects with various data types, the structure comprising 

descriptive elements, each descriptive element having an attribute and attribute value (right 
column on p. 375), e.g. color on Fig. 6, the storage structure further comprising 
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descriptive elements data types including entity or relationship element data type, wherein the 
relationship element data type has associations among entities and its structure includes participating 
entity elements (left column, p. 375) and wherein relationship element data type structure (e.g. Sweep on 
entity-relational graph of Fig. 4) includes an attribute (e.g. Path) (right column, p. 375). 

It would have been obvious to one ordinary skilled in the art at the time the invention was made 
given the suggestions of ISO for defining relationships within description scheme and between 
descriptors Ds including class/object diagrams representing the inter-relationships between the objects 
and their respective features, the teachings of Chang for disclosing an object description model would be 
readily apparent. One would be motivated utilize the teaches of Chang for keeping a variety of complex 
relationships among data in a storage structure, to present all data descriptions, and to retrieve related data 
in an easy manner (left column, p. 372), these descriptions can define new entities or relationships 
between entities with attributes, provide the entity-relationship diagram of the database (schema structure) 
(right column, p. 372). 

Response to Arguments 

9. Regarding claim 25 as amended, it is argued the apphed prior art does not teach claim as recited. 
Specifically, does not teach "non-hierarchical entity relation graph descriptions". 

In response to the above-mentioned argument, applicant's interpretation of the prior art is noted. 
However, it is respectfiiUy noted that according to applicant's specification, broadly speaking object and 
events are used as entities of description (see page 4/lines 16-20), a description is an instantiation of data, 
i.e. an object (see page 7, lines 16-page 8, line 1), organization of multimedia object using the relationship 
among them (page 13, lines 1-8) by modeling general relationships among multimedia object (see page 
14, lines 7-13), spatial relationships among single-media objects inside a multimedia object are described 
using the entity relation graph (page 18, lines 12-25), relationships that cannot be expressed using a 
hierarchical structure (e.g. one object talks to another) are expressed using an entity relation graph, e.g. 
spatial and temporal (i.e. topological or directional) (page 20, line 19-page 23, line 8). Fig. 5 illustrates an 
entity relation graph. 

Therefore, an entity relation graph description is a graphical representation of objects describing 
relationships among them (e.g. temporal/spatial among others) (i.e. non-hierarchical). 

Claimed term related to descriptions, i.e. "non-hierarchical entity relation graph descriptions" has 
been broadly interpreted as general relationships among multimedia objects, e.g. spatial and temporal 
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relationships among single-media objects inside a multimedia object (specs page 20, line 19-page 23, line 
8, wherein Fig. 5 illustrates an entity relation graph). 

Prior art Bergman, teaches a description record to represent content embedded within multimedia 
content (col 7/lines 2-6, embedded hyper-linking, col 15/lines 4-17, spatial relationship of object 
within/contained, col 17/lines 10-19, object within another object, col 17/lines 49-52); Further teaching 
where non-terminal objects include objects with spatial, temporal relationships and thus allows the 
description of these relationships between presentation object (col 3/lines 37-51, see Fig. 5-6), the 
composite object (non-terminal) use graphic block-based representation to describe spatial and temporal 
relationships between the objects (col 5/lines 64-col 6/line 14), The Info Pyramid is a representation 
model of description scheme for describing object the representation includes individual modalities as 
well as additional modalities that include spatial and temporal characteristics, the multimedia content 
description framework or scheme provides a way to represent both spatial and temporal relationships 
among multiple object as well as inter-object user interactions (col 6/lines 39-67), Figs. 5-6 illustrated an 
inter object specification of object with spatial and temporal relationship (i.e. entity relation graph 
description). Thus, prior art teaches a "description record" and an "entity relation graph" as claimed and 
within the scope of applicant's invention. 

10. Regarding claims 25-53 rejected under 103 as being unpatentable over Sezan et. al. in view of 
Bergman, it is argued (p. 7-8) of remarks there is no motivation to combine these references. Specifically, 
because according to applicant, MPEG-7 is still in its formation process, based on this applicant 
concludes that Sezan would not suggest to one of skill in the art to seek technical features outside of the 
current proposed protocol requirements, because this would eliminate compliance with MPEG-7 as it then 
existed. 

In response to the above-mentioned argument, applicant's interpretation of the applied art has 
been fully considered. However, it is respectfully noted that on it face there is found no statement in the 
Sezan reference that states the above assertion. Sezan explicitly states "In the following section, we will 
first describe what MPEG-7 will and will not standardize, i.e. the normative and non-normative parts of 
MPEG-7 (p. 517, left column). "Feature or description extraction algorithms will not be standardized this 
is motivated by the fact that technological advances in such algorithms are expected to materialize in time 
and hence standardizing a current solution will hinder the usability of MPEG-7 in the fiiture, and (ii) such 
algorithms will provide one of the possible means for vendors to differentiate their products. Search 
engines, schemes for database organization for fast and efficient retrieval are also among the non- 
normative parts of MPEG-7 (p. 518, right column). 
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Sezan teaches are not confined to the MPEG-7 requirements. Sezan expHcitly discussing features 
outside of the current proposed protocol requirements, these features (non-normative) will not be in 
compliance with or standardized by MPEG-7, such as feature extraction, because technical advances in 
feature extraction algorithms will hinder the usability of MPEG-7 in the future, suggesting that such 
algorithms will provide possible means for vendors to differentiate their products, and search engine 
designers developing search engines,, schemes for database organization for fast and efficient retrieval as 
among technical features that are not standardized parts of MPEG-7 (p. 518, right column). 

Sezan suggest to one of ordinary skill in the art to seek technical features outside of the current 
proposed protocol requirements, identifying those that would not eliminate compliance with MPEG-7 as 
it then existed. 

11. Regarding claims 25-53 rejected under 103 as being unpatentable over Sezan et. al. in view of 
Bergman, it is argued (p. 8-9) of remarks there is no motivation to combine these references. Specifically, 
because according to applicant, feature hierarchy representation of different features is a requirement in 
the MPEG-7 standard in order that queries may be processing more efficiently in successive layers, 
thereby, Sezan, according to applicant, "makes it clear that they are not exceptions to the hierarchical 
representation 

In response to the above mentioned-argument, applicant's interpretation of the applied prior art is 
noted. Sezan discloses that MPEG-7 will not standardize feature or descriptor extraction algorithms 
unless it is necessary in specifying the descriptor to be standardized. Hence, feature extraction is a non- 
normative part of MPEG-7. As in previous MPEG standards, this is motivated by the fact that (i) 
technological advances in such algorithms are expected to materialize in time and hence standardizing a 
current solution will hinder the usability of MPEG-7 in the future, and (ii) such algorithms will provide 
one of the possible means for vendors to differentiate their products. Search engines, schemes for 
database organization for fast and efficient retrieval are also among the non-normative parts of MPEG-7, 
MPEG-7 will not attempt to accommodate legacy descriptors or description schemes (p. 518, right 
column). 

Sezan discloses that ''search engines, schemes for database organization for fast and efficient 
retrieval are also among the non-normative parts of MPEG-7 '\ thereby queries are not a standardized 
requirement. Thereby, Sezan does not *'makes it clear that they are not exceptions to the hierarchical 
representation" and ''requires to support the hierarchical representation of different features so that 
queries may be processed more efficiently** 
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12. Regarding claims 25-53 rejected under 103 as being unpatentable over Sezan et. al. in view of 
Bergman, it is argued (p. 9-10) of remarks there is no motivation to combine these references. 
Specifically, because according to applicant, Bergman explicitly states that their invention differs fi'om 
the conventional MPEG-7 approach. 

In response to the above-mentioned argument, applicant's interpretation of the applied prior art is 
noted. Bergman's states that his invention has a wide commercial applicability to the exchange of 
multimedia content in general (column 23, lines 54-56). The teachings of Bergman are applicable to 
MPEG-7, specifically, set forth on Example 4, i.e. "A TV News applicafion", disclosed as "applicable to 
illustrate the concepts of InfoPyramid and IPDL, as this is a type of application that MPEG-7 would 
support, it gives an example representation for MPEG-7" (column 22, lines 19-39). Portion cited by 
applicant, should be read in content: 

"In one multimedia application relating to the Motion Picture Experts Group (MPEG) data 
compression standard, for example, a main difference between the InfoPyramid approach of the present 
invention and other conventional schemes proposed for MPEG-7, is the virtually complete elimination of 
the dichotomy between data and metadata. In practice, it has become harder to distinguish between 
transformed data and features. For instance, wavelet coefficients, such as those based on quadrature 
mirror filters and Gabor filters, have been used for both transformations as well as feature extractions. 
Consequently, a data model that can accommodate both data and metadata in a seamless fashion is 
extremely desirable. InfoPyramid accommodates both raw data and transformed data as one of the 
modalities, thus eliminating possible asymmetry introduced by restricting the data model to only 
metadata". Thereby, the InfoPyramid approach of the present invention and other conventional schemes 
proposed for MPEG-7 is different with respect to the dichotomy between data and metadata, Bergman 
identifying deficiencies in the proposed standard invents a data model that can accommodate both data 
and metadata (descriptive data) in a seamless fashion. Portion cited, respectfully, does not state that 
Bergman "teaches away from the proposed standard" not where his teachings are "not applicable to/with 
the proposed standard". 

13. Regarding claims 25-53 rejected under 103 as being unpatentable over Sezan et. al. in view of 
Bergman, it is argued (p. 10-11), that Bergman does not teach generating fi-om multimedia object 
descriptions a non-hierarchical entity relation graph descriptions for one multimedia type. 

In response to the above-mentioned argument, applicant's interpretation of the prior art is noted. 
However, according to applicant's specification, broadly speaking, organization of multimedia object 
using the relationship among them (page 13, lines 1-8) by modeling general relationships among 
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multimedia object (see page 14, lines 7-13), spatial relationships among single-media objects inside a 
multimedia object are described using the entity relation graph (page 18, lines 12-25). 

Bergman teaches on Fig. 3 a block diagram illustrating a preferred data model or description 
scheme (DS) for the multimedia content description framework of the present invention, which includes at 
least one InfoPyramid and one inter-object description model (column 4, lines 11-15); generating a 
modality-fidelity dependency graph (step 1704), including generating descriptions (similar to the 
examples of Figs. 1 1-14) for each of the terminal nodes of the source multimedia content; analyze content 
(step 1 705), decompose the multimedia content source according to the description scheme (as illustrated 
by the examples of Figs. 7, 8, and 10) to extract InfoPyramid representations of each individual media 
modality, the intra- and inter-object relationships', materialize modality and fidelity transformations (step 
1706); based on the usage statistics, those modalities in the modality-fidelity dependency graph are 
preferably materialized by invoking the appropriate modality translation and fidelity transformation 
operators; generate annotations (step 1707); generate necessary annotations of each object, including the 
type, purpose, intention of use, etc. that may be extracted from the original multimedia content. 8. 
Repeat steps 5-7 if the current object is not a terminal object (step 1708) until all the multimedia content 
has been analyzed (column 19, lines 25-colunin 20, line 3). 

14. Applicant's arguments filed 02/22/05 have been fully considered but not rendered persuasive. 

Citation of Pertinent Art: 

15. The following prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Copies of Non-Patent Literature documents cited will be provided as set forth in MPEP§ 
707.05(a): 

A. US 4,479,196 

Ferrer et. al. teaches a "hyperedge" entity-relationship data base systems, a data base model has 
been called the entity-relationship model, wherein this model, which can also be represented by a 
directed graph, the data entities constitute the vertices of the graph while the directed edges of the graph 
represent relationships.between the entities nodes. This model is suitable for some forms of data in which 
data entities and relationships between those data entities are inherent in the real world situation being 
modeled. 
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B. US 5,664,177 

Ha et. al. teach a model for an information cataloging metadata model allows descriptive data (i.e. 
metadata) users to link objects in a linear or peer-to-peer relationship (not hierarchical). The model 
allows a link relationship to the Grouping and Elemental object type categories in a linear or peer-to- 
peer relationship (not hierarchical). One Grouping object or elemental object can have a link 
relationship to many objects in the Grouping or Elemental categories. Thus many-to-many relationships 
can be set up between Grouping and Elemental, Grouping and Grouping, and Elemental and Elemental. 
These relationships allow for the generation of a displayable graph of objects and relationships by 
metadata. 

C. The Metadata Approach of Integrating and Managing Manufacturing Information Systems, Hsu, 
et. al., 1992. 

Hsu et. al. teaches entity relation graph description is a graphical representation of objects 
describing relationships among them (e.g. temporal/spatial among others) (i.e. non-hierarchical). 
Specifically, an Information modeling methodology the combines user's information subjects with their 
contextual knowledge and globally consolidates them into normalized entities and relationships. As 
such, it encompasses the process-oriented functional modeling (e.g. IDEF and Data Flow Diagram) and 
data-oriented semantic modeling (e.g. Object-oriented and Entity-Relationship). 

D. MD^L Content Description of Multimedia Documents for Efficient Process and Search/Retrieval. 
Hu, M.J., School of EEE, Nanyang Technological University. 

Hu, et. al. teaches generating multimedia content descriptions from the multimedia content for the 
content multimedia types; identifying multimedia content, by including more data types and specifying a 
standard set of description schemes and descriptors that can be used to describe various types of 
multimedia information. MPEG-7 will also standardize ways to define and extend different description 
schemes and relationship among these schemes. Such description will then be used to support fast and 
efficient search of audio/visual data and documents such as pictures, graphics, 3D models, audio, speech, 
video. 

E. MPEG-7 Description Scheme for Visual Objects, Ohm, Jens-Rainer, et. al., 1999, p. 153-156. 
Generating multimedia object descriptions from multimedia content for different multimedia 

types, the multimedia type comprising image and video, generating a feature extraction for multimedia 
segments by feature extraction, the feature description including temporal and spatial features. 
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F. Multimedia Description Framework (MDF) for Content Description of AudioA^ideo Documents, 
Hu, M. J, et. al, International Conference on Digital Libraries, Proceedings of the 4^" ACM conference on 
Digital Libraries, Berkley, California, p. 67-75, 2/06/99. 

Hu, et. al. teaches identifying media content by including one or more types of content and 
specifying a set of description schemes that can be used to describe various types of multimedia 
information (i.e. generating multimedia content descriptions from the multimedia content for the content 
multimedia types); parsing objects for features analysis and description abstraction, and storing the object 
description (p. 10, left column), generating a hierarchy tree for the description (p. 10, right column); and 
aggregating different multimedia description schemes, both for MPEG-7 and non MPEG-7 data. At a 
specific viewpoint, an entity can be described by a set of well-defined properties specifying particular 
aspects, characteristics, attributes of the entity, as well as its relationship to other entities 

16. Any inquiry concerning this communication or earlier communications from the examiner should 
be directed to Prieto, B. whose telephone number is (571) 272-3902. The Examiner can normally be 
reached on Monday-Friday from 6:00 to 3:30 p.m. If attempts to reach the examiner by telephone are 
unsuccessful, the Examiner's Supervisor, Andrew T. Caldwell can be reached at (571) 272-3868. Any 
inquiry of a general nature or relating to the status of this application or proceeding should be directed to 
the receptionist whose telephone number is (703) 305-3800/4700. 

Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system, status information for published application may be obtained from 
either Private or Public PAIR, for unpublished application Private PAIR only (see http://pair- 
direct.uspto.gov or the Electronic Business Center at 866-217-9197 (toll-free). 

Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
P.O. Box 1450 
Alexandria, VA 22313-1450 

Hand carried or delivered to: 

Customer Service Window located at the Randolph Bldg. 

401DulanySt. 

Alexandria, VA 22314 

Faxed to the Central Fax Office: 

(703) 872-9306 (old No. in service until Sept. 15, 2005), 
(571) 273-8300 (New Central Fax No.) 



(703) 306-5631 for TC 2100 Customer Service Office. 
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